ENH: IO support for R data files with C extension #41386
Conversation
doc/source/user_guide/io.rst
Outdated
| .. ipython:: python | ||
|
|
||
| rda_file = os.path.join(file_path, "env_data_dfs.rda") | ||
| env_dfs = pd.read_rdata(rda_file, select_frames=["sea_ice_df"]) |
There was a problem hiding this comment.
yeah for read_hdf we do this by forcing the user to have a key. are these always an ordered list? or a keyed list?
|
@jreback - These binary formats (RData, rda, rds) are part of |
|
What is the status here? I see no further response. I can rebase but this ENH may not make it into v1.3. I fully understand that this PR with new C extension is pretty involved and we want to be conservative on large enhancements in |
|
@ParfaitG have been busy but will take a look |
setup.py
Outdated
| if name == "io.rdata._rdata" and is_platform_mac(): | ||
| # non-conda builds must adjust paths to libiconv .h and lib dirs | ||
| include = [ | ||
| os.path.join(os.environ["CONDA_PREFIX"], "include"), |
There was a problem hiding this comment.
Should you check if CONDA_PREFIX is defined? If it isn't, presumably this isn't conda.
There was a problem hiding this comment.
Thanks for this catch. Turns out this if block is not needed. I removed it.
setup.py
Outdated
| @@ -364,6 +370,12 @@ def run(self): | |||
| # https://github.com/pandas-dev/pandas/issues/35559 | |||
| extra_compile_args.append("-Wno-error=unreachable-code") | |||
|
|
|||
| # rdata requires system iconv library | |||
| os.environ["DYLD_LIBRARY_PATH"] = "" | |||
There was a problem hiding this comment.
Overriding the users DYLC_LIBRARY_PATH feels like the wrong solution. Why is this needed? Does this change this path for the duration of the console session, even outside of the setup run?
There was a problem hiding this comment.
This line is not needed and has been removed.
| @@ -364,6 +370,12 @@ def run(self): | |||
| # https://github.com/pandas-dev/pandas/issues/35559 | |||
| extra_compile_args.append("-Wno-error=unreachable-code") | |||
|
|
|||
| # rdata requires system iconv library | |||
| os.environ["DYLD_LIBRARY_PATH"] = "" | |||
| rdata_includes = ["/usr/include"] | |||
There was a problem hiding this comment.
Presumably these should be no on Windows
There was a problem hiding this comment.
Correct. This section is under the if is_platform_mac() condition.
|
If you recall, we explored a Python package for this IO module to read R data files but it had a restrictive license. See previous PR in my first checked item at top. That package was a wrapper to this same C library that we directly interfaced with here, avoiding another soft dependency for pandas. Regarding size, this librdata proclaims itself as a small, lightweight C library. Its source files with two added scripts for Windows totals 177 KB (slightly larger than |
|
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this. |
pandas.read_rdataandDataFrame.to_rdata#40884Proposed RData I/O module interfaces to the C library: librdata.
Overall, this PR includes following changes:
DataFrame.to_rdata)Note: special handling of
iconv, a system resource built-in to Unix machines, is required:iconv.hheader of the GNU C library is included./usr/includeand/usr/lib, which may differ from users installs.iconvis not built-in, two counterpart files (.h and .c script) were added from this repo, win-iconv, where its readme indicates code is placed in the public domain.